System and method for distributed audio recording and collaborative mixing

ABSTRACT

Two or more wireless devices can be independently controlled by their respective users, a mixer component, or a leader wireless device to perform audio recording, convert the recorded audio into a standard or proprietary audio stream format, and transmit the audio stream to a server. The real-time clocks of two or more participating wireless devices can be synchronized. A wireless device can insert timestamps into the audio stream to facilitate the mixing operation. Mixing of the two or more audio streams recorded by wireless devices can be performed by a mixer component either in real time (contemporaneously with the recording) or asynchronously with respect to the recording. The mixing can be performed in a fully automated mode, and/or in an operator-assisted mode.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Utility application Ser. No.12/194,205 filed on Aug. 19, 2008, which claims priority under 35 U.S.C§119(e) of the following provisional application: U.S. Ser. No.60/965,581, filed Aug. 21, 2007, entitled “SYSTEM AND METHOD FORDISTRIBUTED AUDIO RECORDING AND COLLABORATIVE MIXING”, the content ofwhich is incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates generally to wireless devices capable of audiorecording, and more specifically to distributed audio recording andcollaborative mixing by two or more wireless devices.

BACKGROUND OF THE INVENTION

Wireless devices, such as laptop computers, personal digital assistants(PDAs), cellular phones, etc., bring new resources to distributedcomputing. In addition to typical computational resources such as CPU,disk space, and applications, wireless devices increasingly employcameras, microphones, GPS receivers, and other types of sensors. Awireless device by definition has at least one wireless communicationinterface (e.g., cell, radio frequency, Wi-Fi, or Bluetooth™). Usersincreasingly take wireless devices with them to new places, in boththeir personal and professional lives. The ability of wireless devicesto form ad-hoc grids allows using the available resources in acollaborative manner, by aggregating information from the range ofinput/output interfaces found in wireless devices, by leveraging thelocations and contexts in which wireless devices are located, andfinally, by leveraging the mesh network capabilities of wirelessdevices. Wireless grids allow coordinated collaboration of heterogeneousinherently unreliable devices, across unreliable network connections.

The inherent unreliability of wireless devices is primarily caused bythe fact that those devices are, due to their mobile nature,battery-powered. Thus, reducing the power consumption and mitigating theinherent unreliability are two goals of a paramount importance.

Thus, there is a need in distributed systems and applications which canassist in achieving both goals by off-loading processing and datamanagement to non-mobile devices, or to wireless devices which can bereachable with less transmitter power.

SUMMARY OF THE INVENTION

There is provided a system for distributed audio recording andcollaborative mixing by combining audio streams from two or more sourcesinto a single stream that is composed of two or more channels.Leveraging the spatial location of the devices allows the producing ofhigh quality multi-channel sound (e.g., stereo sound or surround sound).

Two or more wireless devices can be located near a sound source, e.g.,at a business meeting, symphony concert, or a live lecture. The wirelessdevices can be independently controlled by their respective users, by amixer component, or by a leader wireless device. The wireless devicescan convert the recorded audio into a standard or proprietary audiostream format, and transmit the audio stream to a mixer component whichcan run on a remote computer.

The real-time clocks of two or more participating wireless devices canbe synchronized. A wireless device can insert timestamps into the audiostream to facilitate the mixing operation.

Mixing of the two or more audio streams recorded by wireless devices canbe performed by a mixer component either in real time (contemporaneouslywith the recording) or asynchronously with respect to the recording. Themixing can be performed in a fully automated mode, and/or in anoperator-assisted mode.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a network level view of a sample embodiment of asystem for distributed audio recording and collaborative mixing

FIG. 2 illustrates the operation of the mixer component in a fullyautomated mode.

FIG. 3 illustrates a sample graphical user interface (GUI) for theoperation of the mixer component in an operator-assisted mode.

The drawings are not necessarily to scale, emphasis instead generallybeing placed upon illustrating the principles of the invention. In thedrawings, like numerals are used to indicate like parts throughout thevarious views.

DETAILED DESCRIPTION OF THE INVENTION

There is provided a system for distributed audio recording andcollaborative mixing, by combining audio streams from multiple sourcesinto a single stream that is composed of multiple channels. Leveragingthe spatial location of the devices allows to produce high qualitymulti-channel sound (e.g., stereo sound or surround sound).

FIG. 1 illustrates a network level view of a sample embodiment of asystem 1000 for distributed audio recording and collaborative mixing.Two or more wireless devices 101 a-101 z can be located near the soundsource 100, e.g., at a business meeting, a symphony concert, or a livelecture. The wireless device 101 can include a central processing unit(CPU), a memory, a wireless communications interface (e.g., cell, radiofrequency, Wi-Fi, or Bluetooth™), a battery, and a microphone. Thewireless device 101 can be provided, e.g., by a cellular phone, apersonal digital assistant, a handheld computer, etc.

The wireless devices 101 a-101 z can have a user interface and/or anapplication programming interface (API) allowing to at least start andstop the audio recording and streaming operations. In one embodiment,the wireless devices 101 a-101 z can be independently controlled bytheir respective users via a user interface. In another embodiment, thewireless devices 101 a-101 z can register with and be controlled by amixer component (not shown in FIG. 1).

The mixer component can run on a remote computer 110. A “computer”herein shall refer to a programmable device for data processing,including a central processing unit (CPU), a memory, and at least onecommunication interface. A computer can be provided, e.g., by a personalcomputer (PC) running the Linux operating system.

Computer 110 can be connected to network 180. While different networksare designated herein, it is recognized that a single network as seenfrom the network layer of the Open System Interconnection (OSI) modelcan comprise a plurality of lower layer networks (e.g., what can beregarded as a single IP network, can include a plurality of differentphysical networks).

In one aspect, wireless device 101 a-101 c can be provided by a PDA andcan connect to network 180 via a wireless access point 114 a-114 z. Inanother aspect, wireless device 101 d-101 z can be provided by acellular phone and can connect to network 180 via General Packet RadioService (GPRS) gateway 150.

The mixer component can transmit control messages to the wirelessdevices 101 a-101 z. The control messages can be encapsulated into,e.g., Blocks Extensible Exchange Protocol (BEEP). The control messagescan include a start recording message and a stop recording command.

Upon receiving a start recording command, the wireless device 101 canactivate its microphone to start recording. In one embodiment, thewireless device can start transmitting the recorded audio stream back tothe mixer component in real time (synchronously with the recording). Inanother embodiment, the wireless device can buffer the audio streambeing recorded and, asynchronously with respect to the recording,transmit the buffered stream back to the mixer component. In a yetanother embodiment, the wireless device can store the recorded audiostream in its memory for later transmission to a mixer component.

Upon receiving a stop recording command, the wireless device 101 mightstop recording audio stream. In one embodiment, the wireless devicemight further stop any synchronous transmission of the audio stream tothe mixer component. In another embodiment, the wireless device 101 canfurther complete any asynchronous transmission of a buffered audiostream to a mixer component.

In another embodiment, the wireless devices 101 a-101 z can elect aleader wireless device which will coordinate the recording by otherparticipating wireless devices. The leader election can be performed,e.g., using an algorithm described in “A Leader Election Protocol ForFault Recovery In Asynchronous Fully-Connected Networks” by M.Franceschetti and J. Bruck, available athttp://caltechparadise.library.caltech.edu/31/00/etr024.pdf.

A skilled artisan would appreciate the fact that any other suitablealgorithm of the leader election can be used without departing from thescope and spirit of the invention.

The wireless devices 101 a-101 z can convert the recorded audio into astandard or proprietary audio stream format, e.g., MPEG-3, RealAudio,Windows Media Audio, etc. The resulting audio stream can be stored bythe recording device locally, and/or transmitted to a remote computer110 via a wireless access point 114 and network 180. Wireless deviceswith no direct connection to wireless access point can leverage the meshnetwork capability of a group of wireless devices, e.g., by establishinga wireless mesh network defined in IEEE 80211s.

In one embodiment, wireless devices 101 a-101 z can have their real-timeclocks unsynchronized. In another embodiment, the real-time clocks oftwo or more participating wireless devices 101 a-101 z can besynchronized using, e.g., Network Time Protocol (NTP) by Network WorkingGroup, available at ftp://ftp.rfc-editor.org/in-notes/rfc1305.pdf. Awireless device can insert timestamps into the audio stream tofacilitate the mixing operation.

Mixing of the two or more audio streams recorded by wireless devices 101a-101 z can be performed by a mixer component (not shown in FIG. 1)running on a remote computer 110. The mixing can be performed either inreal time (synchronously with the recording) or asynchronously withrespect to the recording. Wireless devices 101 a-101 z can also receivethe mixed audio stream back from the mixer, thus allowing the users ofwireless devices 101 a-101 z to listen to the mixed stream.

The mixing can be performed in a fully automated mode, and/or in anoperator-assisted mode.

Operation of the mixer component in a fully automated mode is nowdescribed with reference to FIG. 2. In one embodiment, the mixing can beperformed based upon timestamps included into the audio streams recordedby the individual wireless devices. The wireless device 101 uponreceiving a start recording command from a mixer component at time 210,can transmit to the mixer component a message 210 containing the starttime timestamp, followed by one or more messages 204 containing theaudio stream being recorded. Upon receiving a stop recording commandfrom the mixer component at time 212, the wireless device 101 can stoprecording and continue transmitting the buffered audio stream. Uponcompleting the transmission of the buffered audio stream at time 214,the wireless device 101 can transmit a message 206 containing thetimestamp corresponding to time 212 when it stopped the recording. Thus,the mixer component can use the start time and end time of the audiostream file received for synchronizing it with other audio stream files.The mixer component can also calculate a time stamp for any intermediatepoint of the data stream file by linearly interpolating the start timeand end time timestamps.

In another embodiment, where the real-time clocks of the participatingwireless devices can not be synchronized reliably, the individualrecordings can be synchronized in time based upon one or more clearlydistinguishable events present in all the recordings being synchronized.A clearly distinguishable event can be, e.g., a change in the signalamplitude at a given frequency range where the amplitude level changesby a value exceeding a pre-defined amplitude threshold within a timeperiod not exceeding a pre-defined duration.

The operation of the mixer component in an operator-assisted mode is nowdescribed. Graphical representations of the sound waves over two or moresound channels, e.g., graphs of the audio signal amplitude over time,can be presented to the user via a graphical user interface (GUI) asshown in FIG. 3. The GUI can include two or more graph windows 302 a,302 b. Each of the graph windows 302 a, 302 b can show a waveform graphof an audio signal received from a wireless recording device. The GUIcan further include two or more scroll bars 304 a, 304 b using which auser can scroll the respective graphs 302 a, 302 b along the time axis.The GUI can further have two or more text output fields 306 a, 306 bwhere the timestamp corresponding to the start of the audio streamfragment being displayed in the respective graph window 302 a, 302 b canbe automatically displayed according to the position of the respectivescroll bar 304 a, 304 b within the recorded audio stream file. The GUIcan further have two or more text output fields 308 a, 308 b where thetimestamp corresponding to the end of the audio stream fragment beingdisplayed in the respective graph 302 a, 302 b can be automaticallydisplayed according to the position of the respective scroll bar 304 a,304 b within the recorded audio stream file.

The user can choose a common point of visual distinction (e.g., a pointof rapid signal amplitude change 310 a, 310 b) and align the graphsusing the view slide controls and then pressing the Sync button 320, sothat two or more sound channels are synchronized at the common point 310a, 310 b.

-   -   A small sample of systems methods and apparatus that are        described herein is as follows:        A1. A system for distributed audio recording and collaborative        mixing comprising:

two or more wireless devices capable of audio recording, wherein saidtwo or more wireless devices are located near a sound source to berecorded;

wherein each wireless device of said two or more wireless devices havingan interface allowing at least start and stop audio recording andstreaming operations;

wherein each wireless device of said two or more wireless devices beingconfigured to transmit a recorded audio stream to a mixer component; and

a mixer component configured to combine two or more audio streamsreceived from said two or more wireless devices into a multi-channelaudio stream by synchronizing in time said two or more audio streams,said synchronization being performed based upon one or more clearlydistinguishable events present in all said two or more audio streams.

A2 The system for distributed audio recording and collaborative mixingof A1, wherein said mixer component runs on a remote computer.A3 The system for distributed audio recording and collaborative mixingof A1, wherein at least one wireless device of said two or more wirelessdevices is controlled by a user of said at least one wireless device viaa user interface.A4 The system for distributed audio recording and collaborative mixingof A1, wherein at least one wireless device of said two or more wirelessdevices registers with a mixer component and is controlled by said mixercomponent via an application program interface.A5 The system for distributed audio recording and collaborative mixingof A1, wherein at least one wireless device of said two or more wirelessdevices transmits said recorded audio stream to said mixer componentsynchronously with said recording.A6 The system for distributed audio recording and collaborative mixingof A1, wherein at least one wireless device of said two or more wirelessdevices buffers said recorded audio stream to produce a buffered audiostream, and transmits said buffered audio stream to said mixer componentasynchronously with respect to said recording.A7 The system for distributed audio recording and collaborative mixingof A1, wherein said two or more wireless devices elect a leader device,and wherein said leader device coordinates said audio recording by saidone or more wireless devices.A8 The system for distributed audio recording and collaborative mixingof A1, wherein said synchronizing said two or more recorded audiostreams is performed by an operator via a graphical user interface(GUI), said GUI presenting to said operator two or more graphs of saidfirst audio streams, and allowing said operator to align said graphs atsaid one or more clearly distinguishable events present in said two ormore recorded audio streams.A9 The system for distributed audio recording and collaborative mixingof A1, wherein said synchronizing said two or more recorded audiostreams is performed by a mixer component, said mixer component beingconfigured to synchronize one or more clearly distinguishable eventspresent in said two or more recorded audio streams.B1. A system for distributed audio recording and collaborative mixingcomprising:

two or more wireless devices capable of audio recording,

wherein said two or more wireless devices are located near a soundsource to be recorded, each wireless device of said two or more wirelessdevices having an interface allowing at least start and stop audiorecording and streaming operations, each wireless device of said two ormore wireless devices being configured to transmit a recorded audiostream to a mixer component, each wireless device of said two or morewireless devices having a real-time clock, each wireless device of saidtwo or more wireless devices being further configured insert timestampsinto said

a mixer component configured to combine two or more audio streamsreceived from said two or more wireless devices into a multi-channelaudio stream by synchronizing in time said two or more audio streamsbased upon said timestamps.

B2 A system for distributed audio recording and collaborative mixing ofB1, wherein at least one of said two or more wireless devices isconfigured to synchronize said real-time clock with an external clocksource.B3 A system for distributed audio recording and collaborative mixing ofB1, wherein said mixer component runs on a remote computer.B4 A system for distributed audio recording and collaborative mixing ofB1, wherein at least one wireless device of said two or more wirelessdevices is controlled by a user of said at least one wireless device viaa user interface.B5 A system for distributed audio recording and collaborative mixing ofB1, wherein at least one wireless device of said two or more wirelessdevices registers with a mixer component and is controlled by said mixercomponent via an application program interface.B6 A system for distributed audio recording and collaborative mixing ofB1, wherein at least one wireless device of said two or more wirelessdevices transmits said recorded audio stream to said mixer componentsynchronously with said recording.B7 A system for distributed audio recording and collaborative mixing ofB1, wherein at least one wireless device of said two or more wirelessdevices buffers said recorded audio stream to produce a buffered audiostream, and transmits said buffered audio stream to said mixer componentasynchronously with respect to said recording.B8. A system for distributed audio recording and collaborative mixing ofB1, wherein said two or more wireless devices elect a leader device,wherein said leader device coordinates said audio recording by said oneor more wireless devices.

While the present invention has been particularly shown and describedwith reference to certain exemplary embodiments, it will be understoodby one skilled in the art that various changes in detail may be affectedtherein without departing from the spirit and scope of the invention asdefined by claims that can be supported by the written description anddrawings. Further, where exemplary embodiments are described withreference to a certain number of elements it will be understood that theexemplary embodiments can be practiced utilizing less than the certainnumber of elements.

1. A system for distributed audio recording and collaborative mixingcomprising: two or more wireless devices capable of audio recording,wherein said two or more wireless devices are located near a soundsource to be recorded; wherein each wireless device of said two or morewireless devices having an interface allowing at least start and stopaudio recording and streaming operations; wherein each wireless deviceof said two or more wireless devices being configured to transmit arecorded audio stream to a mixer component; and a mixer componentconfigured to combine two or more audio streams received from said twoor more wireless devices into a multi-channel audio stream bysynchronizing in time said two or more audio streams, saidsynchronization being performed based upon one or more clearlydistinguishable events present in all said two or more audio streams. 2.The system for distributed audio recording and collaborative mixing ofclaim 1, wherein said mixer component runs on a remote computer.
 3. Thesystem for distributed audio recording and collaborative mixing of claim1, wherein at least one wireless device of said two or more wirelessdevices is controlled by a user of said at least one wireless device viaa user interface.
 4. The system for distributed audio recording andcollaborative mixing of claim 1, wherein at least one wireless device ofsaid two or more wireless devices registers with a mixer component andis controlled by said mixer component via an application programinterface.
 5. The system for distributed audio recording andcollaborative mixing of claim 1, wherein at least one wireless device ofsaid two or more wireless devices transmits said recorded audio streamto said mixer component synchronously with said recording.
 6. The systemfor distributed audio recording and collaborative mixing of claim 1,wherein at least one wireless device of said two or more wirelessdevices buffers said recorded audio stream to produce a buffered audiostream, and transmits said buffered audio stream to said mixer componentasynchronously with respect to said recording.
 7. The system fordistributed audio recording and collaborative mixing of claim 1, whereinsaid two or more wireless devices elect a leader device, and whereinsaid leader device coordinates said audio recording by said one or morewireless devices.
 8. The system for distributed audio recording andcollaborative mixing of claim 1, wherein said synchronizing said two ormore recorded audio streams is performed by an operator via a graphicaluser interface (GUI), said GUI presenting to said operator two or moregraphs of said first audio streams, and allowing said operator to alignsaid graphs at said one or more clearly distinguishable events presentin said two or more recorded audio streams.
 9. The system fordistributed audio recording and collaborative mixing of claim 1, whereinsaid synchronizing said two or more recorded audio streams is performedby a mixer component, said mixer component being configured tosynchronize one or more clearly distinguishable events present in saidtwo or more recorded audio streams.
 10. A system for distributed audiorecording and collaborative mixing comprising: two or more wirelessdevices capable of audio recording, wherein said two or more wirelessdevices are located near a sound source to be recorded, each wirelessdevice of said two or more wireless devices having an interface allowingat least start and stop audio recording and streaming operations, eachwireless device of said two or more wireless devices being configured totransmit a recorded audio stream to a mixer component, each wirelessdevice of said two or more wireless devices having a real-time clock,each wireless device of said two or more wireless devices being furtherconfigured insert timestamps into said recorded audio stream; and amixer component configured to combine two or more audio streams receivedfrom said two or more wireless devices into a multi-channel audio streamby synchronizing in time said two or more audio streams based upon saidtimestamps.
 11. A system for distributed audio recording andcollaborative mixing of claim 10, wherein at least one of said two ormore wireless devices is configured to synchronize said real-time clockwith an external clock source.
 12. A system for distributed audiorecording and collaborative mixing of claim 10, wherein said mixercomponent runs on a remote computer.
 13. A system for distributed audiorecording and collaborative mixing of claim 10, wherein at least onewireless device of said two or more wireless devices is controlled by auser of said at least one wireless device via a user interface.
 14. Asystem for distributed audio recording and collaborative mixing of claim10, wherein at least one wireless device of said two or more wirelessdevices registers with a mixer component and is controlled by said mixercomponent via an application program interface.
 15. A system fordistributed audio recording and collaborative mixing of claim 10,wherein at least one wireless device of said two or more wirelessdevices transmits said recorded audio stream to said mixer componentsynchronously with said recording.
 16. A system for distributed audiorecording and collaborative mixing of claim 10, wherein at least onewireless device of said two or more wireless devices buffers saidrecorded audio stream to produce a buffered audio stream, and transmitssaid buffered audio stream to said mixer component asynchronously withrespect to said recording.
 17. A system for distributed audio recordingand collaborative mixing of claim 10, wherein said two or more wirelessdevices elect a leader device, wherein said leader device coordinatessaid audio recording by said one or more wireless devices.